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Abstract 


RACCOON  is  a  vision  systen  that  tracks  car  taillights  at  night.  It  builds  a  global  map  in  real  time 
of  the  lead  vehicle’s  position  based  on  the  location  and  separation  of  the  taillights  in  a  sequence 
of  video  images.  RACCOON  has  been  integrated  into  a  car  following  experiment  on  the  CMU 
Navlab  II,  a  computer-controlled  HMMWV  testbed.  The  Navlab  II  safely  followed  a  lead  vehicle 
on  a  winding  road  in  light  traffic  at  32  km/h. 


1.  Introduction 


Vision  based  autonomous  navigation  has  been  an  active  research  area  for  a  number  of  years 
[Aubert  et  al,  1990]  [Crisman  &  Thorpe,  1990]  [Dickmanns  &  Zapp,  1987]  [Pomerleau,  1990].  Car 
tracking  [Graefe  &  Jacobs,  1991]  [Kuehnle,  1991]  and  car  following  [Kehtarnavaz  et  al,  1991]  have 
also  been  addressed  recently.  However  the  issue  of  caT  following  using  computer  vision  in  low-light 
conditions  remains  unexplored.  At  night,  an  autonomous  land  vehicle  must  deal  with  an  additional 
set  of  problems  including: 


•  Low  light  conditions  make  the  road  difficult  to  see. 

•  Traffic  is  seen  only  as  a  pattern  of  bright  lights  on  a  black  background. 

•  Unlit  landmarks  cannot  be  seen  so  corners  and  intersections  have  to  be  negotiated  based 
solely  on  the  observed  actions  of  the  lead  vehicle. 


Some  of  these  problems  can  be  solved  by  developing  a  system  which  follows  a  human  controlled 
lead  vehicle.  Since  taillights  can  be  easily  extracted  from  a  dark  background,  an  intuitive  approach 
to  the  car  following  problem  is  to  steer  the  autonomous  vehicle  so  that  it  heads  towards  the  taillights 
of  the  lead  vehicle.  This  implementation  may  produce  satisfactory  results  on  straight  roads  when 
both  vehicles  are  moving  at  the  same  speed.  However  this  naive  algorithm  fails  in  any  realistic 
scenario  since  lead  vehicles  make  turns  to  follow  winding  roads,  and  steering  towards  taillights 
results  in  corner  cutting  —  possibly  causing  an  accident  as  the  computer  controlled  vehicle  drifts 
into  oncoming  traffic  or  off  the  road  entirely. 

RACCOON  solves  these  problems  by  creating  an  intermediate  map  structure  which  records  the 
lead  vehicle’s  trajectory.  The  path  is  represented  by  points  in  a  global  reference  frame,  and  the 
computer  controlled  vehicle  is  steered  from  point  to  point.  The  autonomous  vehicle  follows  this 
trail  at  any  desired  speed,  while  keeping  the  lead  vehicle’s  taillights  in  sight.  Using  this  approach, 
the  autonomous  vehicle  steers  around  corners  and  obstacles  rather  than  through  them.  RACCOON 
has  been  successfully  implemented  on  the  Carnegie  Mellon  Navlab  II  testbed  and  follows  a  variety 
of  lead  vehicles  at  night,  on  real  roads,  in  light  traffic.  Its  only  inputs  consist  of  image  sequences 
from  a  single  color  camera  and  Navlab  II  position  information  from  the  onboard  controller. 


2.  System  Architecture 

RACCOON  consists  of  the  following  subsystems:  (see  Figure  1) 

1.  Initialization 

2.  Image  Acquisition 

3.  Taillight  tracking 

4.  Lead  Vehicle  Position  Calculation 

5.  Consistency  Checking  &  Error  Recovery 
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Figure  1:  System  Architecture 
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6.  Output 


Each  of  these  will  be  individually  discussed  below.  RACCOON  can  be  used  in  two  modes: 

1.  Passive  mode:  stand-alone  passive  car  tracking  system. 

2.  Active  mode:  sends  output  to  path  planner  module  for  car  following. 

The  former  mode  enables  RACCOON  to  be  tested  in  the  lab  on  video  taped  image  sequences,  while 
the  latter  may  be  used  either  to  reconstruct  the  path  of  the  lead  vehicle  or  to  actively  follow  the 
lead  vehicle  on  a  real  road. 


2.1.  Initialization 

A  video  image  of  a  night-time  driving  situation  consists  primarily  of  a  black  background  punctu¬ 
ated  by  numerous  bright  spots  and  occasional  specular  reflections  from  metallic  surfaces.  The  lead 
vehicle’s  position  must  be  extracted  from  images  containing  a  large  number  of  taillights.  RAC¬ 
COON  addresses  this  task  by  only  considering  features  which  are  contained  within  a  rectangular 
box  in  the  image  known  as  the  Region  Of  Interest.  By  ignoring  lights  which  lie  outside  this  region, 
processing  speed  is  increased  and  the  possibility  of  tracking  spurious  features  is  decreased.  The 
following  assumptions  are  made  about  the  lead  vehicle: 

•  The  taillights  on  the  lead  vehicle  are  red,  and  bright  enough  to  be  distinguished  from  the 
background. 

•  The  distance  between  the  outermost  taillights  is  known  and  fixed. 

•  The  taillights  of  the  lead  vehicle  will  only  move  small  distances  from  image  to  image. 

Since  the  patterns  of  taillights  on  vehicles  can  vary  tremendously,  RACCOON  makes  no  assump¬ 
tions  about  t’.e  number  of  lights,  their  relative  positions,  or  their  height  from  the  road  surface. 
Any  red  lights  inside  the  region  of  interest  are  considered  to  be  part  of  the  lead  vehicle,  and  thus 
RACCOON  robustly  handles  situations  like: 

•  Illumination  of  turn  signals  or  brake  lights. 

•  Variations  in  relative  camera  pitch  due  to  rough  roads  or  slopes. 

•  Specular  reflections  from  red  cars. 

The  size  of  the  region  of  interest  shrinks  and  grows  as  the  lead  vehicle  changes  distance  relative  to 
the  autonomous  vehicle.  The  increase  in  flexibility  provided  by  this  approach  is  partially  offset  by 
the  danger  of  tracking  spurious  features  which  appear  inside  the  region  of  interest.  The  consistency 
checking  module  attempts  to  recognize  and  correct  for  these  rare  occurences. 

Choosing  which  vehicle  to  follow  is  in  itself  a  difficult  task,  so  the  current  implementation 
of  RACCOON  allows  the  user  to  interactively  select  the  initial  region  of  interest  by  dragging  a 


rectangle  around  the  taillights  of  the  lead  vehicle  in  the  initial  image.  Manual  re-initialization  of 
the  system  is  only  necessary  if  the  lead  vehicle  has  been  irretrievably  lost,  or  if  a  different  lead 
vehicle  is  to  be  followed.  By  default,  the  lead  vehicle  is  expected  to  have  an  approximate  width  of 
1.5  meters,  but  this  can  be  changed  at  run  time  if  the  lead  vehicle  has  unusual  dimensions. 


2.2.  Image  Acquisition 


RACCOON  receives  video  input  as  a  sequence  of  digitized  images  and  operates  identically  on  live 
camera  input,  videotaped  data  or  images  stored  on  disk  in  the  CMU  Generalized  Image  Library 
format  [Hamey  et  al,  1990].  The  rectangular  region  of  interest  is  pre-pTocessed  to  yield  normalized 
red  intensities  at  every  pixel  using  the  formula: 


R 

R  +  G  +  B 


(1) 


Thresholding  is  then  performed  to  keep  only  pixels  which  satisfy: 


•  Absolute  red  intensity  >  T a 

•  Normalized  red  intensity  >  Tb 


where  ^  and  Tb  are  thresholds  specified  at  run-time.  This  selection  process  ensures  that  only 
pixels  which  are  bright  enough  and  red  enough  to  be  taillights  are  considered  by  the  subsequent 
tracker.  The  first  threshold  eliminates  light  sources  which  are  too  dim  to  be  taillights.  The  second 
threshold  measures  the  relative  amount  of  red  content  per  pixel,  and  filters  out  bright  lights  which 
are  not  sufficiently  red,  such  as  head  lights  and  street  lights.  Any  part  of  the  :mage  which  lies 
outside  the  region  of  interest  is  ignored. 


2.3.  Taillight  Tracking 

Within  the  given  region  of  interest,  the  centroid  of  taillight  pixels  and  their  bounding  box  is 
calculated.  The  new  region  of  interest  is  computed  by  expa  ding  the  bounding  box  by  a  fixed 
amount.  This  straightforward  approach  is  significantly  faster  than  more  complicated  algorithms, 
and  has  proven  to  be  adequate  both  in  the  lab  and  on  the  road.  A  model  which  predicts  the  motion 
of  the  region  of  interest  from  image  to  image  was  also  implemented,  but  found  to  be  unnecessary. 

The  distance  to  the  lead  vehicle  and  its  bearing  from  the  autonomous  vehicle  is  calculated  from 
the  horizontal  position  of  the  centroid  and  the  bounding  box  size  as  follows: 

distance  Assuming  that  the  taillights  of  the  lead  vehicle  are  enclosed  in  the  bounding  box,  the 
horizontal  size  of  the  box,  S  is  related  to  the  distance  to  the  lead  vehicle,  D  by  the  formula: 

D  =  kfS  (2) 

where  the  constant  of  proportionality  k  is  pre-computed  from  the  known  camera  parameters 
as  shown  below.  The  vertical  size  of  the  bounding  box  is  expected  to  change  unexpectedly 
with  the  illumination  of  the  third  brake  light  and  is  therefore  ignored. 
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bearing  The  horizontal  position  of  the  centroid  of  the  taillights  in  the  region  of  interest  is  a  good 
indicator  of  the  angle  between  the  lead  vehicle’s  position  and  the  autonomous  vehicle’s  current 
heading.  When  combined  with  the  distance  information,  the  lateral  offset  of  the  centroid  l 
from  the  camera  centerline  enables  us  to  compute  the  vector  to  the  lead  vehicle. 

The  relationship  between  measurements  in  camera  and  image  coordinates  can  be  deduced  from 
Figure  2.  The  following  assumptions  are  implicit  in  the  equations: 


•  The  field  of  view  (FOV)  or  the  focal  length  of  the  camera  is  known. 

•  The  camera  is  not  rolled  relative  to  the  lead  vehicle,  so  that  width  of  the  bounding  box  maps 
to  the  width  of  the  car  in  the  given  image. 

•  The  yaw  of  the  camera  is  known,  and  relatively  small  so  that  any  foreshortening  in  the  lead 
vehicle’s  image  is  negligible. 

•  The  lead  vehicle  is  pointing  approximately  in  the  same  direction  as  the  autom  mous  vehicle 
so  that  foreshortening  in  taillights  is  negligible. 


From  our  model,  we  thus  get  the  following  geometric  relationships: 

=  C  X  |  (3) 

=  Wxh=lxi  (4) 

W 

2  tan  a/2  ^ 

where  L  is  the  lateral  displacement  of  the  lead  vehicle  and  D  is  its  distance  from  the  camera.  The 
intermediate  variable  W  is  the  distance  subtended  by  the  camera’s  FOV  at  the  distance  D. 

Given  the  vector  ( L,D )  to  the  lead  vehicle  in  the  camera’s  reference  frame,  we  can  calculate 
its  position  in  robot  vehicle  coordinates  since  we  know  the  position  and  orientation  of  the  camera 
relative  to  the  computer  controlled  vehicle. 


W 

L 

D 


2.4.  Lead  Vehicle  Position  Calculation 

This  module  is  given  the  position  of  the  lead  vehicle  in  the  coordinate  system  fixed  to  the  robot  ve¬ 
hicle’s  rear  axle.  By  querying  the  controller,  the  current  position  and  orientation  of  the  autonomous 
vehicle  is  acquired.  If  RACCOON  is  being  tested  in  a  lab,  this  is  usually  provided  by  a  vehicle 
simulator;  on  the  Navlab  II  testbed,  the  position  is  computed  by  a  combination  of  dead-reckoning 
and  inertial  navigation  systems.  Using  this  information,  the  module  calculates  the  position  of  the 
lead  vehicle  in  global  coordinates.  If  this  position  is  found  to  be  consistent  with  past  information, 
the  output  module  will  pass  this  position  on  to  other  systems. 


2.5.  Consistency  Checking  &  Error  Recovery 

Occasionally,  the  taillight  tracker  encounters  misleading  inputs  and  will  therefore  report  an  incorrect 
position  for  the  lead  vehicle.  Occasions  where  this  can  occur  include: 
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Figure  2:  Recovering  vehicle  position  vector  from  camera  image 
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•  Mad  image  from  digitizer 

•  Temporary  occlusion  of  taillight(s) 

•  Lead  vehicle  temporarily  out  of  camera  field  of  view 

•  Spurious  red  light  in  region  of  interest 

The  given  lead  vehicle  position  is  compared  with  the  latest  confirmed  position  in  world  coordi¬ 
nates.  and  if  the  Euclidean  distance  between  them  is  within  a  threshold,  the  new  position  is  clssified 
as  being  saiu.  Once  a  number  of  consecutive  sane  positions  have  been  seen,  they  are  additionally 
tagged  as  safe  and  RACCOON  will  send  these  positions  to  the  planner  via  the  output  module.  A 
lone  bad  position  evokes  no  special  response,  but  multiple  consecutive  bad  positions  increase  the 
region  of  interest  to  try  to  reacquire  the  lead  vehicle.  Before  RACCOON  decides  to  reinitialize,  it 
tries  the  following  strategies: 


!.  Searching  a  uniformly  grown  region  ot  interest  at  reduced  resolution. 

Checking  a  large  horizontal  strip  in  the  image,  in  case  the  iead  vehicle  turned  sharply. 

If  the  bounding  box  is  adjacent  to  the  horizontal  image  boundaries.  RA(  (  00. V  advises 
steering  in  that  direction  to  prevent  losing  sight  of  the  iead  vehicle  completely.  1  his  ensures 
that  images  with  clipped  hounding  boxes  are  recognized  early  to  increase  the  chance  of  a 
successful  error  recovery. 

If  ail  of  these  approaches  fail.  RACCOON  can  currently  do  either: 

•  Let  the  user  re-init ialize  the  system. 

•  Find  a  new  set  of  red  lights  by  searching  the  entire  image. 

H.e  latter  approach  will  be  retined  in  later  versions  .'o  that  a  more  intelligent  decision  is  made 
.•.bout  which  set  of  taillights  to  follow:  the  current  implementation  is  sufficient  lor  scenarios  where 
there  are  no  spurious  taillights  in  the  image  ti.e.  single  car  situations). 


2.6.  Output 

Once  the  new  position  has  been  confirmed  as  safe.  RACCOON  reports  the  data  to  the  outside 
worid.  In  stand-alone  mode,  this  consists  of  the  following: 


•  Text  output  to  stdout  which  can  be  piped  (for  logging  purposes).  A  brief  message  is  printed 
as  each  image  is  processed,  and  the  relative  distance  from  the  autonomous  vehicle  to  the  lead 
vehicle  is  shown. 

•  Graphical  output:  A  green  box  is  displayed  surrounding  the  current  region  of  interest.  By 
observing  whether  spurious  lights  have  been  enclosed  by  this  box.  the  user  can  determine 
whether  the  lead  vehicle  is  being  tracked  correctly.  (See  Figure  3). 
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This  distance  can  be  varied  to  change  the  response  of  the  autonomous  vehicle:  in  general,  smellier 
values  result  in  increased  sensitivity,  possibly  leading  to  instabilities,  while  larger  values  give  smooth 
paths,  at  the  expense  of  rounding  sharp  corners.  Good  values  for  this  distance  were  empirically 
determined.  For  this  scenario,  a  value  of  15  meters  was  used. 


4.  Results 

RACCOON  was  first  developed  in  its  passive  mode  and  tested  on  videotaped  images  collected  from 
the  Navlab  II  camera  during  a  human  driven  run.  It  consistently  tracked  the  lead  vehicle  throughout 
the  tape,  detecting  inconsistencies  and  recapturing  the  vehicle  through  auto-initialization  whenever 
necessary.  On  a  Sun-3  equipped  with  frame-grabber  boards,  a  cycle  rate  of  3-5  Hz  was  achieved 
in  debug  mode.  The  time  taken  to  process  a  given  image  was  observed  to  be  approximately 
proportional  to  the  number  of  pixels  in  the  region  of  interest,  thus  inversely  proportional  to  the 
distance  from  the  camera  to  the  lead  vehicle.  The  speed  improvement  for  distant  vehicles  is  balanced 
by  a  corresponding  loss  of  accuracy  since  each  pixel  in  the  image  corresponds  to  a  larger  region  in 
the  world  when  the  vehicle  is  farther  away. 

The  system  was  then  moved  to  the  Navlab  II  testbed  and  recompiled  for  the  Sun  Sparc-2  worsta- 
tion.  As  expected,  the  processing  time  per  image  was  substantially  decreased  and  RACCOON  was 
able  to  process  most  images  at  digitization  rate  (15  Hz).  Since  the  Navlab  II’s  control  bandwidth 
is  only  10  Hz,  processing  speed  was  not  a  problem. 

Initial  tests  in  a  parking  lot  were  promising,  but  highlighted  the  potential  problem  of  losing  the 
lead  vehicle  during  tight  turns  as  it  temporarily  left  the  field  of  view.  The  system  was  modified 
to  be  able  to  handle  situations  where  the  lead  vehicle’s  taillights  are  partially  clipped,  however  a 
complete  solution  to  this  problem  requires  an  orientable  camera. 

Subsequent  test  runs  were  made  on  Schenley  Drive,  a  road  which  offers  hills,  bumps  and  S  curves 
in  addition  to  light  traffic.  The  Navlab  II  followed  the  lead  vehicle  at  approximately  20  mph  (32 
km/h)  on  this  road.  Since  RACCOON  does  not  use  a  flat  earth  hypothesis,  it  is  able  to  handle 
variations  of  the  taillights  in  the  vertical  dimension  of  the  image  caused  by  the  hills  in  the  road. 
RACCOON  was  also  tested  in  light  rain,  and  it  was  observed  that  the  red  reflections  due  to  water 
droplets  on  the  lens  did  not  degrade  performance  since  they  fell  below  the  brightness  threshold  of 
the  taillight  tracking  module. 

One  of  the  advantages  of  RACCOON’S  strategy  is  that  the  computer  controlled  vehicle  need 
not  maintain  a  constant  distance  to  the  lead  vehicle  —  the  path  may  be  followed  at  any  speed. 
Furthermore,  even  if  velocity  control  is  not  accurate,  performance  is  not  affected  since  the  target 
path  remains  valid.  RACCOON  can  also  operate  in  a  decoupled  mode,  where  a  human  driver 
operates  the  throttle,  while  steering  is  performed  under  computer  control. 


5.  Conclusions  and  Future  Extensions 


RACCOON  is  a  system  suited  for  a  specific  domain  (night  time  car  following)  and  it  performs  well 
in  those  conditions.  However,  RACCOON  can  be  integrated  into  existing  systems  as  a  module  for 
a  number  of  other  applications: 
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•  Convoying  operations. 

•  Intelligent  cruise  control  on  highways. 

•  Steering  robot  vehicle  when  the  road  is  ambiguous  or  not  visible. 

•  Safer  driving  in  traffic,  especially  through  intersections. 

Planned  extensions  to  this  project  are  motivated  chiefly  by  shortcomings  of  the  existing  system. 
The  biggest  problem  with  the  current  implementation  is  that  the  single,  fixed  camera  has  a  restricted 
field  of  view  (42.5  degrees)  and  this  is  found  to  be  inadequate  when  the  lead  vehicle  turns  sharply 
and  the  Navlab  II  is  following  closely  (for  example,  at  an  intersection).  Losing  sight  of  the  vehicle  is 
unavoidable  and  this  leads  to  problems  with  reacquiring  the  lead  vehicle  on  completion  of  the  turn 
since  there  may  be  other  cars  in  the  scene.  One  solution  to  this  problem  is  to  mount  the  camera 
on  a  pan/tilt  platform  that  can  be  oriented  to  follow  the  lead  vehicle  during  tight  turns. 

RACCOON  could  be  modified  to  use  active  servoing  of  the  camera  to  maintain  the  lead  vehicle 
in  the  centre  of  its  field  of  view  whenever  possible.  The  transformation  module  would  have  to 
recompute  the  conversion  from  camera  to  autonomous  vehicle  coordinates  from  frame  to  frame  as 
the  configuration  of  the  camera  was  changed. 

A  second  issue  involves  car  following  in  daytime  situations.  Currently,  RACCOON  relies  on 
being  able  to  extract  car  taillights  from  the  background  image  with  very  little  computation  (pixels 
are  classified  as  either  taillight  or  background).  In  the  current  implementation,  this  is  reliable  only 
in  the  evening,  night  or  dawn.  However  alternative  approaches  such  as  optical  flow  or  template 
matching  could  be  used  to  extract  the  position  of  the  lead  car  in  daylight  conditions  and  the 
same  strategy  used  for  following.  These  slower  methods  will  become  more  feasible  once  parallel 
computation  hardware  becomes  available. 

Real-time  perception  systems  are  forced  to  make  certain  concessions  in  robustness  in  order  to 
achieve  the  desired  speed.  Currently  RACCOON  relies  on  other  obstacle  avoidance  systems  (or 
user  intervention)  in  order  to  avoid  collisions  with  unlit  obstacles.  Also  RACCOON  only  tracks  one 
car  in  the  environment  at  any  one  time,  and  this  is  not  sufficient  to  handle  heavy  traffic  conditions. 
Tracking  multiple  objects  using  the  current  hardware  is  not  feasible. 
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